{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Object Detection Inference using Neo-AI-DLR\n",
    "\n",
    "In this example notebook, we describe how to use a pre-trained Object Detection model for inference using the ***Neo-AI DLR*** interface.\n",
    "- The user can choose the model (see section titled *Choosing a Pre-Compiled Model*)\n",
    "- The models used in this example were trained on the ***MS COCO*** dataset because it is a widely used dataset developed for training and benchmarking object detection AI models.   \n",
    "- We perform inference on a few sample images.\n",
    "- We also describe the input preprocessing and output postprocessing steps, demonstrate how to collect various benchmarking statistics and how to visualize the data.\n",
    "\n",
    "## Choosing a Pre-Compiled Model\n",
    "We provide a set of precompiled artifacts to use with this notebook that will appear as a drop-down list once the first code cell is executed.\n",
    "\n",
    "<img src=docs/images/drop_down.PNG width=\"400\">\n",
    "\n",
    "## Object Detection  \n",
    "Object Detection is a popular computer vision algorithm used in many applications such as Person Detection and Vehicle detection. The image below shows object detection results on few sample images.\n",
    "\n",
    "<img src=docs/images/DET.PNG width=\"700\">\n",
    "\n",
    "## Neo-AI-DLR based Work flow\n",
    "The diagram below describes the steps for TVM/NEO-AI-DLR based workflow. \n",
    "\n",
    "Note:\n",
    "   - The user needs to compile models(sub-graph creation and quantization) on a PC to generate model artifacts.\n",
    "       - For this notebook we use pre-compiled models artifacts\n",
    "   - The generated artifacts can then be used to run inference on the target.\n",
    "   - Users can run this notebook as-is, only action required is to select a model. \n",
    "\n",
    "<img src=docs/images/tvmrt_work_flow_2.png width=\"400\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import cv2\n",
    "import numpy as np\n",
    "import ipywidgets as widgets\n",
    "from scripts.utils import get_eval_configs\n",
    "last_artifacts_id = selected_model_id.value if \"selected_model_id\" in locals() else None\n",
    "prebuilt_configs, selected_model_id = get_eval_configs('detection','tvmdlr', num_quant_bits = 8, last_artifacts_id = last_artifacts_id)\n",
    "display(selected_model_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Selected Model: {selected_model_id.label}')\n",
    "config = prebuilt_configs[selected_model_id.value]\n",
    "config['session'].set_param('model_id', selected_model_id.value)\n",
    "config['session'].start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define utility function to preprocess input images\n",
    "\n",
    "Below, we define a utility function to preprocess images for the model. This function takes a path as input, loads the image and preprocesses the images as required by the model. The steps below are shown as a reference (no user action required): \n",
    "\n",
    " 1. Load image\n",
    " 2. Convert BGR image to RGB\n",
    " 3. Scale image\n",
    " 4. Apply per-channel pixel scaling and mean subtraction\n",
    " 5. Convert RGB Image to BGR. \n",
    " 6. Convert the image to NCHW format\n",
    " \n",
    " \n",
    "- Note: The input arguments for the utility function is selected automatically by this notebook based on the model selected from the drop-down menu. \n",
    "\n",
    "- Note: We will be using multiple times in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(image_path, size, mean, scale, layout, reverse_channels):\n",
    "    # Step 1\n",
    "    img = cv2.imread(image_path)\n",
    "    \n",
    "    # Step 2\n",
    "    img = img[:,:,::-1]\n",
    "    \n",
    "    # Step 3\n",
    "    img = cv2.resize(img, (size[1], size[0]), interpolation=cv2.INTER_CUBIC)\n",
    "     \n",
    "    # Step 4\n",
    "    if mean is not None and scale is not None:\n",
    "        img = img.astype('float32')        \n",
    "        for mean, scale, ch in zip(mean, scale, range(img.shape[2])):\n",
    "            img[:,:,ch] = ((img.astype('float32')[:,:,ch] - mean) * scale)\n",
    "    # Step 5\n",
    "    if reverse_channels:\n",
    "        img = img[:,:,::-1]\n",
    "        \n",
    "    # Step 6\n",
    "    if layout == 'NCHW':\n",
    "        img = np.expand_dims(np.transpose(img, (2,0,1)),axis=0)\n",
    "    else:\n",
    "        img = np.expand_dims(img,axis=0)    \n",
    "    \n",
    "    return img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the model using the stored artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dlr import DLRModel\n",
    "\n",
    "model = DLRModel(config['session'].get_param('artifacts_folder'), 'cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform inference\n",
    "\n",
    "### Preprocessing and Inference\n",
    "  - We perform inference on a set of images from the `/sample-images` directory. \n",
    "  - We use a loop to preprocess the selected images, and provide them as the input to the network.\n",
    "\n",
    "### Postprocessing and Visualization\n",
    " - Once the inference results are available, we postpocess the results and visualize the inferred classes for each of the input images.\n",
    " - Object Detection models return results as a list (i.e. `numpy.ndarray`) with a minimum length of 3. Each element in this list contains, the detected object class ID, the probability of the detection and the bounding box co-ordinates.\n",
    " - We use the `det_box_overlay()` function to draw detected boxes on the each object with unique colors for each class ID.\n",
    " - Then, in this notebook, we use *matplotlib* to plot the original images and the corresponding results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from scripts.utils import get_preproc_props\n",
    "\n",
    "# use results from the past inferences\n",
    "images = [\n",
    "    ('sample-images/ADE_val_00001801.jpg', 221),\n",
    "    ('sample-images/ti_lindau_I00000.jpg', 222)\n",
    "]\n",
    "size, mean, scale, layout, reverse_channels = get_preproc_props(config)\n",
    "if type(size) is int:\n",
    "    size = (size, size)\n",
    "print(f'Image size: {size}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tqdm\n",
    "import matplotlib.pyplot as plt\n",
    "from scripts.utils import det_box_overlay\n",
    "from PIL import Image\n",
    "\n",
    "plt.figure(figsize=(20,10))\n",
    "\n",
    "for num in tqdm.trange(len(images)):\n",
    "    image_file, grid = images[num]\n",
    "    img  = Image.open(image_file).convert('RGB')\n",
    "    ax = plt.subplot(grid)\n",
    "    \n",
    "    # Preprocess and run inference\n",
    "    img_in = preprocess(image_file , size, mean, scale, layout, reverse_channels)\n",
    "    res = model.run({model.get_input_names()[0] : img_in}) \n",
    "\n",
    "    img = det_box_overlay(res, img, 0.45, config['postprocess'], img.size, size)\n",
    "    ax.imshow(img)\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot Inference benchmarking statistics\n",
    " - During model execution several benchmarking statistics such as timestamps at different checkpoints, DDR bandwidth are collected and stored. \n",
    " - The `get_TI_benchmark_data()` function can be used to collect these statistics. The statistics are collected as a dictionary of `annotations` and corresponding markers.\n",
    " - We provide the utility function plot_TI_benchmark_data to visualize these benchmark KPIs.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note:</b> The postprocessing part of the object detection model is currently running on ARM, which takes a significant amount of time. In the next release, this would be optimized to run on C7x/MMA which would take something in the order of 1 or 2 milliseconds.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note:</b> The values represented by <i>Inferences Per Second</i> and <i>Inference Time Per Image</i> uses the total time taken by the inference except the time taken for copying inputs and outputs. In a performance oriented system, these operations can be bypassed by writing the data directly into shared memory and performing on-the-fly input / output normalization.\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from scripts.utils import plot_TI_performance_data, plot_TI_DDRBW_data, get_benchmark_output, print_soc_info\n",
    "stats = model.get_TI_benchmark_data()\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(10,5))\n",
    "plot_TI_performance_data(stats, axis=ax)\n",
    "plt.show()\n",
    "\n",
    "tt, st, rb, wb = get_benchmark_output(stats)\n",
    "\n",
    "print_soc_info()\n",
    "print(f'{selected_model_id.label} :')\n",
    "print(f' Inferences Per Second    : {1000.0/tt :7.2f} fps')\n",
    "print(f' Inference Time Per Image : {tt :7.2f} ms')\n",
    "print(f' DDR usage Per Image      : {rb+ wb : 7.2f} MB')"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
